Adaptive packet routing

ABSTRACT

A method of determining the latency of a route in a packet-switched network, a packet switch for use in such a method and network and a packet-switched network are disclosed. Preferably, each switch maintains a routing table that records the latency of the routes accessible by that switch. Each switch also preferably has a GPS-based universal time clock which it employs to time the transmission and arrival of identifiable timing packets, these times being used to compute route latency and to up-date the routing tables. In one example ( FIG. 1 ) a packet-switched network has a plurality of switches (S 1 -S 6 ) interconnected by links or trunks (T 1 -T 7 ). A local GPS-base clock (GPS CLK) is connected to each switch (S 1 -S 6 ) to enable the accurate timing of transmission and reception of identifiable timing packets in accordance with a system-wide universal timing standard.

TECHNICAL FIELD

This invention relates to means for or associated with adaptive routingof packets, cells, data, messages or sessions in telecommunicationsand/or computer networks covering large geographical areas. It alsorelates to network management systems for such networks employingadaptive routing.

The term ‘adaptive routing’ indicates that the route taken by packetsacross the network takes account of varying conditions within thenetwork, such as delays, faults, congestion, etc. The invention issuited for use in packet-switched networks that usehop-by-hop/packet-by-packet routing protocols (for example IP orInternet Protocol) with store-and-forward techniques, as well as innetworks where the complete route is negotiated at the start of asession and maintained throughout the session (for example ATM orAsynchronous Transfer Mode protocols).

The invention is also concerned with means for determining link and/orrouter latencies in a network and for using such determinations toup-date and maintain latency and/or routing tables in the network.

BACKGROUND TO THE INVENTION

Common and long-established methods for assessing the quality of a linkin an IP network involve the use of ICMP [Internet Control MessageProtocol] to send echo-reply control messages between network entitiesto check if a remote device is accessible and the delays involved. Thismethod commonly made use of a program, PING [Packet Internet Groper], tocheck accessibility and transmission delays. The resultant data was thenused to maintain routing tables for dynamic hop-by-hop routing usingSNMP [Simple Network Management Protocol] over TCP/IP [TransmissionControl Protocol/Internet Protocol].

Though still in widespread use, ‘pinging’ has a number of seriousdrawbacks as a means for determining link and router latencies (packettransit and processing delays). First, the receipt and processing of theping and the generation of the echo-reply by the recipient requires avariable amount of processing time, dependant upon the instantaneousprocessing load of the recipient. Second, the receipt and processing ofthe echo and the computation of the transit time requires a variableamount of processing time at the sender, which also depends upon theinstantaneous processing load of the sender and adds to the apparentlatency of the network elements involved. Third, a ping that involvesmultiple hops via multiple routers using a hop-by-hop protocol may notbe indicative of the following message or session transmission speedsince—in a TCP/IP system—many of the packets comprising the messagemight take different routes to their destination as a result of dynamicrouting. Moreover, the interrogation and echo packets of a ping maythemselves take different routes in a hop-by-hop routing system. Fourth,pinging using hop-by-hop protocols will poorly represent the latency ofan end-to-end connection established under the ATM protocol since theroute negotiated for the ATM message is likely to be different to thattaken by the ping. Fifth, pining is highly wasteful of network resourcesbecause:

-   -   (i) each network element must be pinged from each other element        to establish and maintain a comprehensive set of routing tables,    -   (ii) though a control packet carrying a ping or echo may transit        many intermediate nodes or routers, those intermediate devices        cannot garner latency information from that packet, and    -   (iii) each ping involves two transits across the network, one        interrogation and one echo.

It is known for each router to maintain a router table or database thatrecords at least some qualitative property of each link connected to arouter to inform routing decisions; that is, to allow the mostappropriate intermediate node or router to be selected for theforwarding of a packet or for establishing an end-to-end ATM connection.In many networks, the network manager maintains a master router table ordatabase that records all such information for the network. Since thecapacity and transmission delay (latency) of a route will vary accordingto traffic load and the capabilities of the links and routers in thatroute, router tables need constant updating. This is especiallyimportant for the connection-oriented ATM protocol where transmissionquality ‘contracts’ are negotiable. The use of constantly updatedrouting tables at each router for the purpose of dynamically determiningrouting in a network is often called ‘adaptive routing’.

While latency is one of the most important quality parameters of a link,it is not the only one and it is known to record other fixed andvariable characteristics of a link in a router table. For example,International patent application PCT/SE98/02345 [WO 99/33232] byEricsson discloses an algorithm for computing a quality parameter called‘link cost’, recording that parameter in router tables and using it forestablishing connections in an ATM system. The inputs for thiscomputation are themselves computed variables such as maximum celltransfer delay, peak-to-peak cell delay variation, available cell rate,cell loss ratio, and the like. While link latency is a vital input forsuch computations, the Ericsson application is silent with regard to howlink latency is determined.

U.S. Pat. No. 4,771,424 to Hitachi discloses an adaptive routing systemin which the queue length at a router on a link is used as a proxy forthe latency of that link and router tables are maintained using thatdata. However, there are two important problems with this: first, thelatency of the link itself is ignored so that a slow link thatterminates at a fast receiver with no packet queue is seen to have zerolatency and, second, interrogation/management packets must wait theirturn in the queue for processing and return with the requested data.

U.S. Pat. No. 5,805,602 to Bell Atlantic Network Services, Inc disclosesa method of minimising packet jitter when decoding a stream of ATM cellscarrying audio or MPEG-encoded data into an IP packet stream, whichmethod uses both the internal program clock reference [PCR] carried bythe audio or MPEG data and an external clock at the receiver. However,this patent is not concerned with adaptive routing and there is nosuggestion that packet jitter might be used as a proxy for link quality,or that either the variation between the PCR time and ‘absolute’ time(eg, UTS time) might be used in that way. However, this Bell Atlanticpatent rightly emphasises that special care is needed when transmittingvideo or audio packets and, by implication, highlights the need foraccurate and up-to-date data on composite route latency in a networkwhen routes are being negotiated for such packets. Nevertheless, theBell Atlantic patent is silent as to how either link or router latencyis measured.

In a large and complex network—such as the global Internet—there may bemany intermediate switches and links/trunks in a virtual circuit and thechoice of route can be an important factor in end-to-end communicationsspeed, even where all relevant switches and links are operatingnormally. Routing algorithms are known and widely used but tend to becomputationally intensive and are generally unable to take account ofrapidly varying local traffic conditions, or to handle data according topriority or bandwidth requirements. They also involve considerable costin terms of management overheads if link latency is measured by pingingor by a proxy such as packet queue length at router ports. Thealternative expedient of setting aside trunk and switch capacity tohandle streaming video/audio data, or other high-priority data, isinefficient and costly in terms of the utilization of network capacity.

OUTLINE OF INVENTION

From one aspect the present invention comprises a method of determiningthe latency of a route between two switches in a packet-switchedtelecommunications network. In this method, an identified timing packetis transmitted from a first switch to a second switch in the network viaa predetermined route and the universal time of transmittal of thepacket at the first switch is determined by the use of a first clock ator near the first switch, which clock receives timing signals from asystem of earth satellites. [For convenience, such packets may behereafter referred to as ‘timing packets’ or ‘identified packets’ aswell as ‘identified timing packets, and such clocks may be referred toas ‘GPS clocks’.] The time of transmittal of the first packet isrecorded at the first switch and the universal time of its receipt isrecorded at the second switch is recorded using a GPS clock at or nearthat switch. The times of transmission and receipt of the identifiedpacket are then used to determine the latency of the route and to updaterouting table(s) at the switches and/or at a network controller ormonitor.

The time of transmittal can be recorded in the timing packet itself,read by the receiving switch(es) and the latency of the route thencomputed by the receiving switch(es). Alternatively, the transmittingswitch and the receiving switch(es) can interrogate one another to findout the respective receipt and transmission times of the identifiedpacket so that each can then compute the latency of the route and updateits own routing table. Alternatively, a master switch or controllercould under take the interrogation, make the computation and then informeach of the relevant switches of the latency of the route and/or varioussections of it.

A convenient way of recording the time of transmittal and receipt of anidentified packet is for the transmitting switch to set a flag in thepacket as it is being transmitted (or, less preferably, immediatelybefore transmission) and to record the time at which the flag is set asproxy for the time of transmission. Similarly, the receiving switchdetects and resets the flag in the identified packet and records thetime of resetting as proxy for the receipt time. For this to beeffective, it is desirable for the flag to be in the packet header in alow-level protocol.

From another aspect, the invention comprises a switch suitable for usein a packet-switched telecommunications network, the switch having anassociated GPS clock and means to record the universal time oftransmission and/or reception of identified timing packets via multipleroutes to and from other switches in the network. The switch may alsohave means for storing a routing table and means for referencing thattable when deciding the route on which to send normal packet traffic.The switch may then have means for calculating the latency of routes byreference, inter alia, to the recorded universal times of transmissionand receipt of identified packets.

From another aspect the invention may comprise a digitalpacket-switching telecommunications network including a plurality ofswitches (including exchanges, routers, servers and the like) connectedto one another by one or more telecommunications links/trunks in whichat least one switch automatically maintains a routing table listing thespeed or latency of links, other switches and/or routes comprisingcombinations of links and switches, the routing table being used by theswitch (or by a network manager or ATM sender) as an input whendetermining the routing of packets or messages; the network beingcharacterised in that:

-   -   each switch includes or is associated with a GPS-based clock,        and    -   the transit times, as measured by the use of the GPS clocks, of        packets received or despatched by the switch are employed to        up-date the routing table at the switch.

A preferred method of determining packet transit times and updatingrouting tables makes use of a GPS-based event recorder of the typedisclosed in our Australian patent 716041. However, any event recordermay be used that is able to access a standard accurate time that isuniversal or common across the network and satellite-based clocks thatdo not have the features disclosed in our prior patent can be used inthe present invention. Most usually, the universal or common time willbe UTS time and the most common way of accessing that time will be viatime signals distributed globally by satellite systems such as the GPSset of satellites.

It is preferred, for the purposes of this invention, to make use of thefact that packets associated with network management (as well as manyother packets) tend to include unique identifying numbers as well asaddress codes indicative of the sender and the intended recipient(whether it be the initiating sender and the ultimate destination orhop-by-hop intermediate senders and receivers). These packets mayinclude low-level headers that indicate the types of packet (eg, atiming packet). Provision may be made for a timer flag (a single bitwill usually be all that is needed) to be included in or near the packetheader so that it can be readily recognised and changed.

The coding of a packet as a timer packet with the identifyingtransmitter and receiver addresses, together with the insertion of anydesired administrative commands or information into the timer packet,can be done by the sending switch in advance of transmission, but thetimer flag should remain unset. The timer packet can then be queued fortransmission on a selected network route in the normal manner. Only whenthe timing packet is actually being placed on the route, is the timerflag set and that event recorded as proxy for the time of transmission.

It is preferable for the timer flag or the packet type identification tobe made at or near the network-protocol level so that the minimum ofprocessing is needed before the packet can identified as a timer packetand its identification number (‘ID’) can be read. Of course, the timerpacket may include the intended destination address, the sender'saddress, a datagram inserted by the sender for interpretation by theaddressee using common protocols such as TCP/UDP/IP. For example, theaddressed switch may be requested to return data for recent timerpackets sent, or it may be provided with data for updating its routertable.

Immediately upon receiving a packet, recognising it as a timer packetand determining its ID, receiving switch can reset the timer flag andthat event is used as proxy for packet reception so that the eventrecorder at the receiving switch can accurately identify the time ofreceipt. The time of receipt, along with the packet ID and any otherdata required (such as the identity of the sending switch) can then berecorded in a data log maintained by the event recorder. Even if thetimer packet does not include a timer flag, the type of packet can bequickly identified and the time of packet receipt can be determined withminimum of processing at the receiving switch. After recording the timeof receipt of the packet, the packet can be added to the queue forprocessing by the switch. If it is to be forwarded to another switch, anew header (containing the new address along with the packet ID and atimer flag) can be applied and the packet placed in the queue fortransmission. Again, the event recorder can record the time oftransmission and packet ID at substantially the same time as the packetis being placed on the network. In this way, router latency can beaccurately and separately measured independently of link latency.

Of course, if the received timer packet is intended for the receivingswitch (as determined by the destination address), it will need to befurther processed at higher protocol levels at that switch so as todetermine whether there is a message or instruction that is intended foran application process running at the receiving switch. The destinationswitch will not normally be required to forward that timer packet toanother switch. Perhaps the most important an immediate function of thetimer packet will have been performed by causing the time of its receiptalong with its identity to have been recorded as an event in the evenrecorder of the destination switch.

When the sender/originator of the timer packet in the above scenarioneeds to up-date its routing table with the latency data associated withthe timer packet or packets that it has transmitted, it can send anormal network management packet requesting the receipt times for one ormore identified timer packets at one or more of the switches enroute toone or more destinations (including the destination switch or switches,if desired) to which it has sent timer packets. Upon receipt of theinterrogation packet, each relevant reporting switch reads the recordedtime(s) of receipt and/or transmission for each identified timer packetand transmits that data, along with the reporting switch's identity, tothe sender/requester. The sender/requester can then automaticallycompute the latency for the respective link and/or switch by subtractingthe recorded time of transmittal of the timer packet from the reportedtime of receipt, and then use that information to update its routertable information.

While this method of updating router tables can be employed by each orany switch in the network, an alternative method is also envisagedwhereby the overheads associated with timer packets can be reduced. Anintermediate switch that receives and forwards a timer packet can useits event recorder to record not only the time of receipt and the packetID, but also the identity of the originator, the identity of theultimate destination and, possibly, the identity of the upstream switchfrom which the packet was received. This allows the intermediate switchto send query packets to the other identified switches to requestinformation about the time that the timer packet was sent and received,permitting the intermediate switch to update its router table withoutitself having to send a timer packet. Alternatively, even greateroverhead reductions can be achieved where a master switch (the networkmanager) is the only one to send timer packets and to request returns.In that case, the master switch can then send the latency informationgathered to other switches in the network, which then update theirrouter tables accordingly.

Another method of determining packet transit times in accordance withthe present invention is for the sending switch to use its GPS clock totime-stamp selected outgoing packets with the time of despatch and forthe receiving switch to determine the time of receipt of the packetusing its GPS clock and to then compute and record in its routing tablethe latency of the relevant link or links and/or the latency ofcommunication with the sending switch. This has the advantage that itneed only involve a single short packet sent one-way, but it has thedisadvantage that, in most implementations, the transit time recordedwill include variable amounts of packet processing time at the senderand receiver switches. As in the first described method, thetime-stamped network management packets need only be sent during periodsof light traffic so that the routing tables of switches can be updatedwithout impinging upon traffic throughput.

Since each switch in a network can keep its own latency records in thisway, simple routing algorithms can be employed to ensure the fastestavailable transmission when a virtual circuit is established or beingnegotiated. Thus congested or slow links can be automatically avoided.Alternatively, where a circuit of pre-designated priority is beingnegotiated, a link with the most appropriate latency can beautomatically selected by a sending switch by simply referring to itslook-up table. In this way, a high-priority session that is intended tocarry live packets or other high-speed data can be assigned to the linkwith the lowest latency; conversely, a low priority session intended fornormal email or file transfer can be assigned to the slowest links,leaving the faster links available for higher priority data.

It will be appreciated that a route between two subscribers cannot beestablished solely by local selection of the fastest link at eachswitch. Attention must also be paid to the packet's/session's ultimatedestination. Accordingly, overall route selection will normally be underthe general control of network routers. If desired, the router caninterrogate each switch that may be relevant for the circuit todetermine the latency of its associated links. Link selection can bemade by the router routine rather than at the switch level.

From another aspect, the present invention involves methods forimplementing the above-indicated routing procedures. One methodcomprises the step of building tables of link latency at each of aplurality of switches within a telecommunications network by readingtime-stamped packets sent by other connected switches, computing thelapsed time between the time stamp and the time of receipt to determinethe latency of the relevant link, recording that link's latency in therelevant table, and looking up the table when a message is to betransmitted to determine the appropriate link for use in thetransmission. The method may include the step of matching recorded linklatency with the relative priority of the message or session in order toassign the appropriate link for transmission. The method may include thestep, at each switch, of deriving a GPS-referenced clock signal for thepurpose of time-stamping outgoing timer packets, and for the purpose ofdetermining the time of receipt of an incoming timer packet.

It should be noted that the term ‘latency’ as used herein is anindication of the speed of a link, or a series of links and switches.Latency can be graded according to slowness or speed. It should also benoted that the use of the initials ‘GPS clock’ is intended to encompassor refer to any timekeeper that is periodically calibrated, reset orotherwise automatically adjusted by the use of satellite-derivedsignals. It is not intended to refer uniquely or exclusively to the setof US satellites that are commonly called the ‘global positioningsystem’ or the ‘global positioning satellites’.

DESCRIPTION OF EXAMPLES

Having broadly portrayed the nature of the present invention, examplesof the implementation of the invention will now be indicated by way ofillustration only. The examples will be described with reference to theaccompanying drawings in which:

FIG. 1 is a diagram of a group of interconnected switches that formspart of a telecommunications network of the first example.

FIG. 2 is a diagram showing one of the switches of the network of FIG.1, together with its connecting links, in more detail.

FIG. 3 is a diagram illustrating the structure of a timer packetemployed in the first example.

FIG. 4 is a flow diagram illustrating the procedure by which a timerpacket is generated and accepted by interconnected switches of the firstexample.

FIG. 5 is an example of a look-up table that is automatically built andmaintained at each switch of the network for the purpose of intelligentrouting.

FIG. 6 is a schematic diagram of a switch of the second example thatsends a timer packet, receives a reporting packet and updates itslatency or router table.

FIG. 7 is a schematic diagram of an intermediate switch or router of thesecond example that transmits and receives timer and/or reportingpackets.

FIG. 8 is a schematic diagram of a destination or receiver switch of thesecond example that receives timer packets and generates reportingpackets on demand.

FIG. 1, illustrates a small portion of a large packet-switchingtelecommunications network comprising a group of digital switches S1-S6in connected together via trunks or links, some of which are identifiedas T1 to T7.

Each trunk or link itself is normally divided into many separatecommunications channels and many separate messages or sessions arenormally carried by each channel using known multiplexing techniques.The trunks or links normally extend over substantial distances (tens orhundreds of kilometres) and may employ a variety of transmissionmedia—microwave beams, optical fibres and/or wire (whether coaxial ortwisted pairs). They will normally include a number of intermediaterelay or amplifier stations to amplify and reconstitute attenuated ordistorted signals.

It will be appreciated that a physical connection, such as an opticalfibre or a train of microwave stations, may carry a plurality of‘virtual’ trunks and that a channel may be encoded to serve as aplurality of virtual sub-channels. The terms indicate stages in thecomplexity of multiplexing. It is sufficient for the present purposes tonote that every switch is connected to a neighbouring switch by a largenumber of parallel channels so that there is a wide choice betweenchannels for any particular session. Also, where heavy traffic isexpected between any two switches, the switches will be interconnectedby a large number of parallel channels that will normally be groupedinto a plurality of physical or virtual trunks. This is diagrammaticallyillustrated with switches S3 and S5 which are directly interconnected byparallel trunks T1 and T2 and switches S5 and S6 which are connected byanother pair of parallel trunks T3 and T4. The through-put speed(inverse of latency) of each trunk and channel can vary considerablydepending upon the state of the physical media and the performance ofthe relay stations.

For convenience of description, it will be assumed that each trunk hasonly four multiplexed channels (identified as C1-C4). This configurationof trunks and channels is illustrated diagrammatically for switch S3 inFIG. 2. Thus, when establishing a route for a session (ie, when settingup a virtual circuit), a particular channel of a particular trunk mustbe selected. Normally, the selection of parallel trunk T2 or T3 and ofthe particular channel (C1 to C4) will be left to a predeterminedroutine built into the transmission equipment that operates withoutregard to varying latency, session priority or data type. This ‘blind’or ‘dumb’ trunk and channel allocation is replaced by the ‘intelligent’routing method of the present invention.

It will be apparent, however, that the matter of overall route selectionis more complex than the selection of an appropriate one of a number ofalternative parallel channels between switches. A large number ofalternative indirect connections are available. For example, switch S3could be connected to switch S5 indirectly via S2 or S4, less indirectlyvia S2 and S4, even less directly via S1, S2 and S4, etc. Nevertheless,the methods of the present invention will be useful in assisting theoverall routing function as they can be applied between any pair ofswitches in the network (not merely adjacent switches).

In the chosen example, a local GPS-referenced clock GPS CLK is connectedto each switch so as to ensure that all switches in the network areprovided with a common time-reference signal having a precision of theorder of microseconds and, preferably, of the order of nanoseconds. InFIG. 2 the letters ‘GPS’ identify the GPS receiver modules while theletters ‘CLK’ identify the associated clock circuits. The GPS and clockcircuits may be formed as disclosed in our above-mentioned internationalpatent application.

When one user at terminal U1 connects to its local ISP1 via localexchanges E1 and E2 to establish a session with a second user at remoteterminal U2 via the second user's local ISP2 and its local exchanges E3and E4, ISP1 sends the request to the nearest switch S1 and the sessionis set up so that S1 is connected to S6 (the switch closest to ISP2).The general route employed between S1 and S6 maybe determined byexisting routers using (i) regional maps (tables) that indicate the mostdirect (or otherwise most preferable) route between S1 and S2, (ii)current information about intermediate switches or links that are out ofservice and, if desired, (iii) data on trunk channel speeds provided bythe system of the present invention. Otherwise, channel and paralleltrunk selection can be left for automatic allocation at the switch levelon the basis of a fixed ‘try-sequence’ based upon an overall networkmap, the priority level allocated to the session and the switch look-uptables that are maintained (in accordance with the present invention) ateach switch on the attempted route.

Of course, before the main session is established with U2, user U1 can‘ping’ ISP1 and check the two-way speed of the last kilometre and localswitch S1. Indeed, user U1 can ping user U2 in a preliminary testsession to check the likely speed of connection to U2. However, theroute that is later established for the main session may not be the sameas that established for the ‘ping session’ and the results are likely tobe deceptive. It is, of course, impractical for a router to effectivelyping all possible routes between two users in order to establish thatwhich provides the desired speed or priority. The administrative burdenon the system would be grossly excessive and connection times wouldoften be unacceptably long.

In accordance with the chosen example, each switch automaticallygenerates and sends occasional timing packets to each neighbouringswitch (and, optionally, to other indirectly-connected switches). Thefrequency of such packets can vary as desired with the nominal speed ofthe relevant link, packets being sent at intervals of the order oftenths of seconds to minutes as desired. A typical timing packet P isillustrated diagrammatically in FIG. 3, while the procedure by whichsuch packets are generated and handled between switches S3 and S5 isindicated by FIG. 4. FIG. 5 shows the form of the look-up tablegenerated and maintained at switch S3 relating to trunks T1, T2, T3, T6and T7.

In general, conformity with the Internet protocol, the timer packetgenerated by S5 for transmission to S3 includes the address of thedestination switch (S3 in this case) in its header, followed by thesource address (of S5), and then followed by a data segment comprising(i) the trunk and channel being tested (say, T2-C3) and (ii) thetime-stamp or tag applied by the sender (S5). Finally, the packetconcludes with a cyclic redundancy check code CRC and an appropriatepacket end code (not indicated in FIG. 3). The procedure for generatingpacket P at S5 and for processing the received packet at S3 is generallyindicated by the flow chart of FIG. 4.

At the appropriate interval, the creation of a timer packet P isinitiated within S5 so as to incorporate (i) the address of destinationC3, (ii) the identity of the trunk and channel to be employed and (iii)the precise time at which the packet is created as read from the GPSclock. The packet is then multiplexed onto the appropriate trunk andchannel (T2-C3) and transmitted to S3. The channels of T2 aredemultiplexed at S3 and the address destination and type of each packetis determined. Immediately packet P is recognised as a timer packetaddressed to S3, S3 reads its clock GPS CLK and computes the timedifference between transmission and receipt, so determining the latencyof T2-C3. The resultant data is recorded in a look-up table of the typeindicated in FIG. 5 at switch S3. The data in the look-up table is thenused at a later time to determine the most appropriate trunk and channelto employ for a session having a given class of priority.

It will be appreciated, however, that switch S6 could also generate atimer packet addressed to S3 via a particular channel in trunk T4 or T5and another in trunk T2 or T3, and that the table at T3 could beextended to record the overall latency of such a multiple ‘hop’ (theidentity of both channels employed being included in the timer packet).In theory, each switch could maintain a record of the latency of allchannels and all combinations of channels in a network in this mannerand the routing of a session would be determined by reference to thetable held in the first switch to be encountered. For example, switch S1is the first switch for outgoing messages from ISP1 and U1.

As an alternative, a high-level router could be used to determine themost direct possible route in the conventional manner and theninterrogate all switches on that route to determine if a channel of therequisite speed is available for each link. If so, the session can beset up accordingly. If not, all switches on the next most direct routeare interrogated.

The second example, shown in FIGS. 6-8, illustrates an alternative meansof updating routing or latency tables in the routers of a network usinga flag incorporated in the header of a packet, marking that packet as atiming packet. In this example, the flag is preferably incorporated inthe network header that is normally stripped off the packet immediatelyupon receipt by a router, before it is queued for processing. However,the flag may be incorporated in a higher-level header, such as the IP orUDP header, or a flag may be incorporated in more than one of thepacket's headers. For convenience, it will be assumed in this examplethat the flag is incorporated in a header that also includes some meansof identifying the packet, such as a sequence number. Thisidentification will be referred to as the packet ID.

In the example of FIGS. 6-8 one router 10 of a network 12 of otherrouters R1, R2 . . . initiates a process for updating information in itsrouter table 14 by generating a short timing packet 16 that needcomprise little more than a header incorporating a destination address,a timing flag and a unique packet ID. It will be assumed that packet 16is addressed to destination router R4 and is to be routed via routersR1, R2 and R3 using ATM protocol. Router 10 incorporates a GPS-basedevent recorder 18 of the type disclosed in our prior patent. Timingpacket 16 is created in a process diagrammatically indicated at 20 andqueued for transmission in queue 22. When packet 16 is being transmittedits timer flag is detected and the packet ID is immediately input toevent recorder 18, which records the UTS time with the ID as a packettransmission event in its event table 24.

Upon receipt of packet 16 at router R1 [indicated at 25 in FIG. 7], thepacket header is immediately read and the packet recognised as a timerpacket, whereupon the packet ID and UTS time are immediately recorded inthe event table 26 of R1's event recorder 28 as a packet receipt event.Since (by pre-arrangement under the ATM protocol) packet 16 is to beforwarded to R2, it is processed at 30 and added to the packet-transmitqueue 31 of R1. When packet 16 is being transmitted to R2, its header isagain read and, upon detection of the timer flag, the packet ID and theUTS time are recorded in event table 26 of recorder 28 as a transmissionevent. The process of recording receipt and transmission eventsassociated with packet 16 is then repeated in successive routers R2 andR3.

Finally, immediately upon receipt of timer packet 16 at destinationrouter R4 [indicated at 34 in FIG. 8], the header of the packet is readand recognised as a timer packet and the time of receipt and the packetID are recorded in the event table 36 of the event recorder 38associated with router R4. The packet is placed in the receiving queue40 for processing in unit 42. In the event that no other data orinstructions are contained in the timer packet, it is destroyed as it isnot to be forwarded to any other router, router R4/34 being thedestination stipulated by router 10 at the time the route for timerpacket 16 was set up.

It will be appreciated, as previously indicated, that any router maysend a timer packet to any other router to which it is connected by asingle hop. Whether and how frequently this can be done will be a matterfor regulation by the network manager.

When network manager or router 10 wishes to collect latency data toup-date its router table 14, it can send a normal and separate networkmanagement query packet to each of routers R1, R2, R3 and R4 requestingtheir respective receipt and transmission times for packets with IDsspecified by manager 10.

Alternatively, a single query packet may be routed through each routerin turn along the route taken by the initial timer packet 16. Similarly,the manager may require the return of a separate data packet containingthe desired time information from each router, or it may require eachrouter to add its own time information to a common packet as it is beingreturned from the destination router R4. In any event, the time datareturned by a router will include that router's ID as well as thereceipt and transmission times for each packet ID specified by networkmanager 10.

For the sake of illustration, it will be assumed that router 10generates a query packet 16 a that does not have its timer flag set andis addressed to destination router R4/34 containing a message requestingrouter R4/34 to return its time data for the packet with the ID ofpacket 16. The receipt of packet 16 a is not recorded in the eventtimers 18, 28 and 38 of routers 10, 25 and 34 (respectively) because itstimer flag is not set. Upon receipt at destination router 34, packet 16a is treated as a normal management packet by process 42, whichgenerates a data request 44 addressed to event timer 38 for the time ofreceipt of a packet with the ID of packet 16. This data (indicated at46) is incorporated in a packet that is put on the network at thetransmit unit 48 and (for the sake of example) is addressed to router 10via intermediate routers R3, R2 and R1. If packet 16 a also containsdata for use by router R4 in updating its router table 50, the updatinginformation is incorporated in table 50 as indicated by path 52 (shownin broken lines).

The returning packet from 34/R4 is received and processed by R3 asindicated at 56 and a request for timing data associated with packet 16is generated at 58 and passed to event recorder 28, which outputs dataat 60. In this example, data at 60 is added to that obtained from R4 andput on the network addressed to router 10 via intermediate routers R2and R1. Since R3 is able to compare the time it recorded for thetransmission of packet 16 with the time that R4 reported for the receiptof packet 16, it is able to compute the latency of the relevant link andto up-date its router table 62, as indicated by path 64 shown in brokenlines. Additionally or alternatively, table 62 can be up-dated from theinformation in returning packet placed there by network manager 10 asshown by path 66 shown in broken lines. Finally, the returning packet istransmitted to manager 10 via transmitting unit 68.

At the network manger 10, the returning packet is received and processedat 70, the down-stream timing data is extracted at 72, a request isgenerated at 74 for the transmission data from event recorder 18pertaining to timer packet 16 and the latencies for each link and eachintermediate downstream router (R1, R2 and R3) are computed at 76. Usingthis latency data, the master router table 14 is updated.

Assuming, as will normally be the case, that the downstream routers havenot directly updated their router tables in a piecemeal fashion, router10 can (at an appropriate time) generate a multicast or broadcast packet(indicated at 80 in FIG. 6), which is then distributed so as to effectthe updating of all relevant routing tables at much the same time.

Thus, to recapitulate, the router tables of routers other than thenetwork manager 10 in network 12 can be updated in various ways. First,the network manager can have the sole responsibility for initiatingtimer packets, collecting timing data, computing latency information forthe network and up dating the router table of each router in thenetwork. This will have the advantage of low network overhead and highuniformity of router tables, but may place a burden on the manager suchthat the intervals between updating of all network routing tables isexcessive. Alternatively, each intermediate router involved in amulti-hop transmission of a timer packet and involved in the samemulti-hop response to a request for latency data can partially up-dateits router table by extracting latency information from the query packetbeing returned. This partial up-date could be enhanced if themanager/initiator were to include its transmission time for eachidentified packet in its query. Alternatively again, each router caninitiate its own timer packets, call for timing data from any otherrouter(s) and update the latency information in its router tableindependently of the network manager or any other router.

It will be appreciated that this use of GPS-based event recorders of thetype disclosed in our prior patent provides a particularly detailed andpowerful method of maintaining router tables in a network. It permitsthe latency of any link and any router to be efficiently and accuratelymeasured at any time, and it permits the frequent updating of any or allrouter tables in a network, with the minimum of management overheads.The need for inherently approximate, inaccurate and burdensome proxies(such as packet queue length) for router latency is eliminated. Instead,highly accurate (to the microsecond or better, if desired) link and/orrouter latencies can be determined with little burden upon routers ornetwork managers.

While the benefits of the invention are evident from the abovedescription of the chosen examples, it will be appreciated that manychanges and modifications can be made without departing from the scopeof the present invention as outlined above. For example, as alreadynoted, it is not essential for there to be a separately identifiableflag in a timer packet where a distinctive header for timer packets isemployed or where the header has a ‘type’ field that can be used toidentify a timer packet. However, it is desirable to permit a timerpacket to be identified with the minimum of disassembly or processing bya router to ensure that the minium of router processing latency is addedto the up-stream link latency.

1. A method of determining the latency of a route between two switchesin a packet-switched telecommunications network, comprising:transmitting an identified packet from a first switch to a second switchin the network via a predetermined route, wherein the identified packetcomprises a timing flag to indicate that the identified packet is atiming packet, recording the universal time of transmittal of saidpacket at the first switch by the use of a first clock at the firstswitch that receives timing signals from a system of earth satellites,recording the universal time of receipt of said packet at the secondswitch by the use of a second clock at the second switch that receivestiming signals from said system of earth satellites, and employing saidtimes of transmission and receipt of said identified packet to determinethe latency of said route.
 2. A method according to claim 1 furthercomprising: encoding the universal time derived from the first clockinto one of said identified packets transmitted by the first switch tothe second switch via a predetermined one of multiple routes, saidencoded time being the time of transmittal of said one packet, readingthe encoded time in said one packet when it is received at the secondswitch, and using the read encoded time, together with the time ofreception of said one packet at the second switch, to determine thelatency of said predetermined route from the first switch to the secondswitch.
 3. A method according to claim 1 further comprising: setting thetiming flag in one of said identified packets transmitted by the firstswitch substantially simultaneously with the transmission of said onepacket to the second switch via a predetermined route, recording at thefirst switch the universal time, having regard to the first clock, atwhich said timing flag is set, examining packets received at the secondswitch to determine if each is one of said identified packets and, ifso, whether said timing flag has been set, resetting the timing flag insaid one identified packet at the second switch substantiallysimultaneously with the reception of that packet at the second switch,recording at the second switch the universal time, having regard to thesecond switch, at which said timing flag is reset, and comparing theuniversal time at which said timing flag was set in the first switchwith the universal time at which that timing flag was reset at thesecond switch to determine the latency of said predetermined route.
 4. Amethod according to claim 3 further comprising: transmitting adata-request packet from the first switch to the second switchrequesting the universal time at which the timing flag in said oneidentified packet was reset, receiving at the first switch saiddata-request packet from the second switch, reading at said first switchthe universal time at which the timing flag in said one identifiedpacket was reset, computing at the first switch the latency of saidpredetermined route by comparing said time read from the data-requestpacket with the universal time recorded at the first switch, andupdating a routing table stored in a memory at the first switch withsaid computed latency.
 5. A method according to claim 3 furthercomprising: transmitting a data-request packet from a network controllerto the first switch requesting the universal time at which the timingflag in said one identified packet was set transmitting a data-requestpacket from said network controller to the second switch requesting theuniversal time at which the timing flag in said one identified packetwas reset, receiving at the network controller said data-request packetsreturned from the first and second switches, computing at the networkcontroller the latency of said predetermined route by comparing theflag-set and flag-reset times read from said data-request packets inrespect of said one identified packet, and updating a routing tablestored in a memory at the network controller with said computed latency.6. A method according to claim 3 wherein said predetermined routebetween the first and second switches is via an intermediate thirdswitch, the method further comprising: identifying at the third switchthe source, namely the first switch, and the destination, namely thesecond switch, of said one identified packet as it is passed through thethird switch en route to the second switch, generating and transmittinga first a data-request packet from the third switch to the first switchrequesting the universal time at which the timing flag in said oneidentified packet was set, generating and transmitting a seconddata-request packet from said third switch to the second switchrequesting the universal time at which the timing flag in said oneidentified packet was reset, receiving at the third switch saiddata-request packets returned from the first and second switches,computing at the third switch the latency of said predetermined route bycomparing the flag-set and flag-reset times read from said data-requestpackets in respect of said one identified packet, updating a routingtable stored in a memory at the third switch with said computed latency,and transmitting portions of the routing table at the third switchrelevant to the first and second switches to said first and secondswitches respectively.
 7. A method according to claim 6 wherein thethird switch comprises a network controller.
 8. A method according toclaim 1 wherein the timing flag comprises a single bit within a packetheader of the identified packet.
 9. A method according to claim 8wherein the packet header of the identified packet comprises a networkheader of the identified packet.
 10. A method according to claim 1wherein the identified packet comprises multiple headers, wherein morethan one of the multiple headers of the identified packet incorporatesthe timing flag.
 11. A packet switch for use in a packet-switchingnetwork, the switch having: connection means to connect the switch to aplurality of telecommunications links, transmitter means for placingidentifiable outgoing timing packets on to one or more of said links,wherein the timing packets comprise timing flags, receiver means fortaking identifiable incoming packets off one or more of said links,clock means to derive universal time from the signals of a system ofearth satellites, recording means for recording the identity of each ofsaid outgoing and each of said incoming packets together with theuniversal time of transmission and reception, respectively, of eachpacket, memory means to store in a routing table data signifying thelatency of network routes accessible from each of said links, andupdating means to compute the latency of said routes by reference to thetimes of transmission and reception, as derived by said clock means, ofsaid identifiable packets, and said updating means to store such latencydata in the memory means in a manner that effects the updating of saidrouter table.
 12. A packet switch according to claim 11 to processincoming packets generated by remote switches, which incoming packetsinclude time-data indicative of the universal time at the respectiveremote switch where said incoming packet was generated, the packetswitch including: encoding means associated with said transmitter meansto encode the universal time at the packet switch corresponding to thetime of transmission of each of said outgoing packets and to insertencoded time-data into each packet that is representative of the time oftransmission of that packet, decoding means associated with saidreceiver means to decode the time-data in each of said incoming packetsand to feed the decoded time-data for each respective incoming packet tosaid updating means.
 13. A packet switch according to claim 12 toprocess incoming packets generated by remote switches, each of whichincoming packets includes a timing flag, wherein: said transmitter meansto set a timing flag in each of said outgoing packets at the time oftransmission of said packet, said receiver means to detect the timingflag in each incoming packet and to reset that timing flag, saidrecording means to record the time at which said timing flag is set inan outgoing packet as proxy for the time of transmission of that packet,and wherein said receiver means to record the time that the timing flagof an incoming packet is reset as proxy for the time of reception of therespective incoming packet.
 14. A packet switch according to claim 11 toprocess incoming packets generated by remote switches, each of whichincoming packets includes a timing flag, wherein: said transmitter meansto set the timing flag in each of said outgoing packets at the time oftransmission of said packet, said receiver means to detect the timingflag in each incoming packet and to reset that timing flag, saidrecording means to record the time at which said timing flag is set inan outgoing packet as proxy for the time of transmission of that packet,and wherein said receiver means to record the time that the timing flagof an incoming packet is reset as proxy for the time of reception of therespective incoming packet.
 15. A packet switch according to claim 11wherein each of the timing flags comprises a single bit within a packetheader of the corresponding timing packets.
 16. A packet switchaccording to claim 15 wherein multiple packet headers of at least one ofthe timing packets comprise the timing flag of the corresponding timingpacket.
 17. A packet switch according to claim 15 wherein the packetheader comprising the single bit also comprises a packet identifier ofthe corresponding timing packet.
 18. A packet switch according to claim11 wherein the timing flag comprises a value other than a value of time.19. In a packet-switched telecommunications network: a first switch totransmit and receive packets via multiple routes in the network to andfrom a second switch, memory means to store a routing table comprisingdata indicative of the latency of each one of said multiple routesbetween said first and said second switches, logic means to consult saidrouting table and select one of said multiple routes for a packet to betransmitted from the first switch, having regard to the latency of saidroute stored in the routing table, a first clock at the first switch toderive universal time from the signals of a system of earth satellitesand a second clock at the second switch to derive universal time fromsaid system, first recording means at the first switch to record thetime of transmission and reception of identified packets by said firstswitch having regard to the universal time derived by said first clock,wherein the identified packets comprise timing flags to indicate thatthe identified packets are timing packets, second recording means at thesecond switch to record the time of transmission and reception ofidentified packets by said second switch having regard to the universaltime derived by said second clock, and updating means to determinelatency of each one of said multiple routes, having regard to therecorded times of transmission and reception of said identified packets,and to update said routing table with said determined latencydeterminations.
 20. A packet-switched telecommunications networkcomprising: a plurality of packet switches connected to one another by aplurality of telecommunications links to form multiple routes forpackets travelling from source to destination in the network, memorymeans in each switch to store a routing table comprising data indicativeof the latency of each one of a plurality of possible packet routes fromthat switch to a destination, logic means in each switch to consult therespective routing table and to select a route for a packet to betransmitted from that switch having regard to the latency of said routestored in the respective routing table, a clock associated with eachswitch to derive the universal time at the associated switch from thesignals of a system of earth satellites, recording means at each switchto record the time of transmission and reception of identified packetsat that switch having regard to the universal time at that switch asdetermined by the clock associated with that switch, wherein theidentified packets comprise timing flags to indicate that the identifiedpackets are timing packets, and updating means to update the routingtable of a switch by reference to the recorded times of transmission andreception of said identified packets within the network.